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SUMMARY 

Technology-based  systems  for  education,  training,  and  performance  aiding  (including  decision  aiding) 
may  pose  the  ultimate  test  for  validating  approaches  to  integrate  humans  with  automated  systems. 
These  systems  need  to  model  students  and  users.  The  models  they  generate,  as  well  as  the  interactions  based 
on  them,  must  adapt  to  the  evolving  knowledge  and  skills  of  individual  students  and  users.  Evaluation  findings 
suggest  that  such  adaptations  are  feasible,  worthwhile,  and  cost-effective.  Data  drawn  from  many  evaluations 
of  technology-based  education  and  training  indicate  overall  that  these  systems  can  reduce  costs  by  one-third 
and  that  they  can  additionally  either  reduce  the  time  to  achieve  instructional  objectives  by  one-third  or 
increase  achievement  (holding  time  constant)  by  one-third.  The  likely  impact  on  military  readiness  and 
effectiveness  suggested  by  these  findings  is  significant.  Similar  results  indicating  increased  personnel 
effectiveness  and  cost-savings  have  been  found  in  evaluations  of  technology-based  performance  aiding 
systems.  They  suggest  a  need  to  determine  and  re-adjust  the  balance  between  resources  allocated  to  training 
and  resources  allocated  to  performance  aiding  systems.  Development  of  sharable,  reusable  objects  and 
capabilities  for  assembling  these  objects  on  demand  and  in  real  time  will  substantially  increase  accessibility 
and  reduce  costs  of  education,  training,  and  performance  aiding  while  making  them  asynchronously  and 
continuously  available  -  regardless  of  distance  and  time.  Specifications  and  capabilities  for  such  objects  are 
the  goals  of  much  research  and  development.  Some  of  these  goals  are  discussed  under  the  systems 
engineering  categories  of  analysis,  design  and  development,  delivery  and  management,  and  evaluation. 
This  research  agenda  would  be  substantially  enhanced  by  NATO/PfP  participation,  which  might  include 
development  of  a  NATO/PfP  directory  of  data-bases  permitting  wide  dissemination  and  sharing  of  techniques 
and  findings  and  the  development  of  NATO/PfP  common  practices  for  the  development  of  sharable  objects 
and  guidelines  for  their  use. 


INTRODUCTION 

Continuous  adult  learning  is  the  hallmark  of  establishing  and  maintaining  military  occupational  competence 
and  expertise.  This  learning  environment  must  increasingly  rely  on  technology  in  order  to  be  continuously 
accessible  to  learners  and  other  users.  Because  of  the  autonomous  nature  of  technology,  such  learning 
environments  will  place  enormous  responsibility  on  developers  to  design  and  implement  materials  from  the 
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perspective  of  the  learner  (Figure  1).  Education,  training,  and  performance  aiding  systems  may  be  the  ultimate 
test  bed  for  validating  technological  approaches  to  accommodate  humans  in  automated  systems. 


Figure  1:  Dimensionality  of  Human  Roles  in  Training  Systems. 


Automated  instructional  systems  must  address  a  number  of  variables  including  the  method  of  instructional 
delivery,  desired  learning  objectives,  and  learning  environment.  Figure  1  illustrates  these  elements  in  three- 
dimensions. 

•  The  “Complexity  of  Technology/Technique”  axis  provides  a  continuum  of  technical  difficulty 
beginning  with  Computer  Aided  Instruction  and  progressing  in  difficulty  to  a  Virtual  Reality 
Distributed  Simulation.  Educational  objectives  and  availability  of  required  equipment  define  the  level 
of  technology  required  for  instruction. 

•  The  “Eevel  of  Eeaming”  axis  concerns  the  level  of  knowledge  required  by  the  task(s)  to  be 
performed.  Benjamin  Bloom’s  taxonomy  offers  a  convenient  framework  for  defining  levels  of 
learning  from  basic  knowledge  to  the  more  involved  process  of  evaluation.  (Bloom,  1956). 
These  levels  suggest  the  instructional  techniques  to  be  used  and  affect  the  training  time  needed  to 
reach  them. 

•  The  “Environment”  axis  denotes  where  the  information  learned  will  be  used,  ranging  from  a  familiar 
(e.g.,  office)  environment  to  the  unfamiliar,  complex  environments  characteristic  of  military 
operations.  Environment  determines  what  training  devices  will  be  available  and  prioritizes 
instructional  objectives. 

A  key  point  of  this  paper  is  that  there  are  many  roles  that  each  human  will  play  in  interactions  with  any 
system,  including,  as  Figure  1  suggests,  training  systems.  The  figure  suggests  that  these  roles  must  evolve 
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along  with  the  level  of  learning  aehieved  by  individuals.  Capable  systems  (notionally  the  numbered  items  in 
Figure  1)  must  aeeommodate  these  levels  of  learning  dynamieally  as  their  users  aehieve  them,  adjusting 
both  the  eomplexity  of  technologies  and  techniques  they  employ  and  the  environment  they  present  to  learners. 
To  be  effective  in  providing  continuous  adult  learning,  both  the  system  and  its  users  must  change  their 
behavior  -  both  must  adapt  and  learn. 

Can  we  expect  this  to  happen?  Will  technology,  specifically  computer  technology,  adapt  to  its  users  in 
much  the  same  way  that  users  today  must  adapt  to  it?  The  technology  is  certainly  becoming  more  powerful. 
After  almost  40  years,  Moore’s  law  (e.g..  Service,  1996)  still  holds. 

In  1965,  Gordon  Moore  noted  that  engineers  were  doubling  the  number  of  electronic  devices  (basically 
transistors)  on  chips  every  year.  In  1975,  he  revised  his  statement  to  say  that  the  doubling  was  occurring  every 
two  years.  If  we  split  the  difference  and  predict  it  will  occur  every  18  months,  our  expectations  fit  reality  quite 
closely. 

One  consequence  of  Moore’s  law  is  that  significant  computational  power  and  functional  capacity  are  found 
today  on  desks  and  in  laps  for  less  than  $1000  per  system.  Moore,  however,  did  not  predict  the  pace  of 
progress  in  human- centered  design  of  computer  software  interfaces  or  in  software  applications.  This  rapid 
increase  in  electronic  capability  was  discussed,  ten  years  ago  by  Raymond  Kurzweil,  writing  about  The  Age 
of  Intelligent  Machines  (1992)  and,  more  recently,  about  The  Age  of  Spiritual  Machines  (2000)  in  which  he 
predicted  that  a  $1000  unit  of  computing  will  equal  the  computational  capability  of  the  human  brain  by  the 
year  20 1 9  -  and  exceed  it  thereafter.  Whether  or  not  this  $  1 000  unit  will  be  “intelligent”  and  have  functional 
capability  adaptively  centered  on  the  human  user  can  be  debated,  but  the  rapid  expansion  of  computational 
capability  into  many  areas  of  human  cognition  seems  undeniable. 

The  pervasive  nature  of  automation  technology  and  applications  to  military  enterprises  seems  equally 
undeniable.  Not  only  present  in  our  offices,  schools,  and  homes,  it  has  become  an  essential  enabling  capability 
for  our  military  operations.  We  seem  inexorably  home  on  a  rising  crest  of  digital  applications  in  command 
and  control,  communications,  modeling  and  simulation,  sensors  and  surveillance,  and  -  notably  -  in  such 
human-intensive  activity  as  decision-making,  education,  training,  and  performance  aiding.  These  latter 
applications  and  their  inherent  technology  challenges  are  the  topic  of  this  paper,  which  addresses  the 
following  questions: 

•  Is  it  worth  it?  What  do  we  know  about  the  value  of  technology  applications  in  learning  (education 
and  training)  and  performance  aiding  (decision  making  and  job-task  enhancement)  and  what  are  the 
challenges  to  realizing  the  full  value? 

•  Where  do  we  want  to  so?  What  are  promising  technological  opportunities  and  directions  for  these 
applications  and  how  might  we  best  employ  them  to  support  military  operations  and  activities? 

•  How  should  we  set  from  here  to  there?  How  can  science  and  engineering  advance  the  state  of  the  art 
and  the  state  of  our  practice  in  performing  military  operations? 


IS  IT  WORTH  IT? 

We  will  argue  “yes”.  Scientists  and  engineers  have  been  pursuing  research  and  development  on, 
and  assessments  of  computer  applications  in  education,  training,  and  performance  aiding  for  more  than 
45  years  (Fletcher  and  Rockway,  1986).  Much,  possibly  most,  of  this  work  in  the  United  States  was  funded 
by  the  military  (Congress  of  the  United  States,  1988)  and  was  intended  to  enhance  military  capabilities. 
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This  pattern  of  funding  support  may  be  true  for  other  eountries  as  well  (Chatelier  and  Seidel,  1993;  Seidel  and 
Chatelier,  1995).  It  is  not  surprising,  then,  to  find  that  data  have  been  aeeumulating  on  the  effeetiveness  and 
eost  benefits  of  these  applieations.  We  might  well  wish  for  more  data,  partieularly  in  the  area  of  eost  benefits, 
but  the  weight  of  evidence  has  advanced  to  the  point  that  it  is  much  easier  to  argue,  on  the  basis  of  existing, 
empirical  data,  for  the  value  of  these  applications  than  against  them.  These  data  are  reviewed  in  the  following 
two  sub-sections  on  education  and  training  applications  and  on  performance  aiding  applications. 

Education  and  Training  Applications 

Suppose  we  could  reduce  the  time  it  takes  for  military  personnel  to  reach  a  given  level  of  training  by 
30  percent.  We  would  then  expect  to  see  reductions  in  expenditures  for  instructor  pay  and  allowances, 
student  pay  and  allowances,  temporary  duty  costs,  training  equipment  costs,  and  installation  support  costs  - 
among  the  measurable  things.  Not  surprisingly,  we  find  that  savings  from  the  reduction  of  training  time  can  be 
substantial  -  a  “value”  that  may  go  beyond  the  measurable  (e.g.,  flexibility  in  the  reuse  of  the  recovered  or 
“saved”  time). 

For  instance,  the  United  States  military  spends  about  $4  billion  a  year  on  specialized  skill  training.  This  is 
the  training  needed  after  “basic”  or  accession  training  to  qualify  personnel  for  the  many  technical  jobs 
(e.g.,  wheeled  vehicle  mechanics,  radar  operators,  avionics  technicians,  medical  technicians)  needed  to 
perform  military  operations.  It  does  not  include  the  costs  of  aircraft  pilot  training,  which  comprise  a  separate 
cost  category.  Figure  2  shows  the  annual  reductions  in  costs  that  would  result  if  instructional  time  were 
reduced  by  30  percent  for  20,  40,  60,  and  80  percent  of  the  US  military  personnel  who  complete  specialized 
skill  training  each  year.  For  instance,  if  the  US  were  to  reduce  by  30  percent  the  time  to  train  20  percent  of  the 
personnel  undergoing  specialized  skill  training,  it  would  save  over  $250  million  per  year.  If  it  were  to  do  so 
for  60  percent  of  the  personnel  undergoing  specialized  skill  training,  it  would  save  over  $700  million  per  year, 
an  appreciable  amount  by  almost  any  standard. 


Figure  2:  Monetary  Savings  in  Speciaiized  Skiii  Training  with  30  Percent  Reduction  in  Training  Time. 
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Time  savings  of  30  percent  are  used  here  for  a  reason.  As  Table  1  below  suggests,  savings  of  about  this 
magnitude  in  training  time  are  frequently  found  in  reviews  of  instructional  technology.  Orlansky  and  String 
(1977)  reported  that  reductions  in  time  to  reach  instructional  objectives  averaged  about  54  percent  in 
their  review  of  technology  used  in  military  training.  Fletcher  (1991)  found  an  average  time  reduction  of 
3 1  percent  in  6  assessments  of  interactive  videodisc  instruction  applied  in  higher  education.  Kulik  reported 
time  reductions  of  34  percent  in  17  assessments  of  technology  used  in  higher  education  and  24  percent  in 
15  assessments  of  adult  education  (Kulik,  1994).  All  these  reviews  were  independent  in  that  they  reviewed 
different  sets  of  evaluation  studies.  On  this  basis,  it  does  not  seem  unreasonable  to  expect  instructional 
technology  applications  to  reduce  the  time  it  takes  students  to  reach  a  variety  of  given  instructional  objectives 
in  military  education  and  training  by  about  30  percent. 


Table  1 :  Percent  Time  Savings  for  Technology-Based  Instruction 


Study  (Reference) 

Number  of 
Studies  Reviewed 

Average  Time 
Saved  (Percent) 

Orlansky  and  String  (1977) 

13 

54 

Fletcher  (1991) 

8 

31 

Kulik  ( 1 994)  (Higher 

Education) 

17 

34 

Kulik  ( 1 994)  (Adult  Education) 

15 

24 

For  that  matter,  30  percent  is  a  conservative  target.  Commercial  enterprises  that  develop  technology-based 
instruction  for  the  Department  of  Defense  (DoD)  regularly  base  their  bids  on  the  expectation  that  they  can 
reduce  instructional  time  by  50  percent  (Fletcher,  personal  communication).  Noja  (1991)  has  reported  time 
savings  through  the  use  of  technology-based  instruction  as  high  as  80  percent  in  training  operators  and 
maintenance  technicians  for  the  Italian  Air  Force. 

Where  do  these  time  savings  come  from?  To  a  large  extent  they  may  be  accounted  for  by  individualization 
of  pace  -  the  speed  with  which  students  move  through  instructional  material  and  reach  instructional 
objectives.  Even  the  most  rudimentary  of  technology-based  instruction  systems  are  found  to  adjust  pace  for 
individual  students. 

Does  the  capability  to  adjust  pace  matter?  Many  classroom  instructors  have  been  struck  by  the  differences 
in  the  pace  with  which  their  students  learn.  Their  observations  are  confirmed  by  research.  For  instance, 
consider  the  following  findings  on  the  time  it  takes  for  different  students  to  reach  the  same  instructional 
objectives: 

•  Ratio  of  time  needed  by  individual  hearing  impaired  and  Native  American  students  to  reach 
mathematics  objectives:  4  to  1  (Suppes,  Fletcher,  and  Zanotti,  1975;  1976) 

•  Overall  ratio  of  time  needed  by  individual  students  to  learn  in  grades  K-8:  5  to  1  (Gettinger,  1984) 

•  Ratio  of  time  needed  by  undergraduates  in  a  major  research  university  to  learn  features  of  the  LISP 
programming  language:  7  to  1  (Private  communication,  Corbett,  1998) 

We  may  not  be  surprised  to  discover  differences  among  students  in  the  speed  with  which  they  are  prepared  to 
learn,  but  the  magnitudes  of  the  differences  do  seem  surprising.  As  we  might  expect  from  Gettinger’ s  1984 
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review,  a  typieal  Kindergarten  through  Grade  8  elassroom  will  have  students  who  are  prepared  to  learn  in  one 
day  what  it  will  take  other  students  in  the  same  elassroom  5  days  to  learn.  This  differenee  does  not  seem  to  be 
mitigated  by  more  homogeneous  grouping  of  students  based  on  their  abilities.  The  students  in  Corbett’s 
(1998)  research  university  are  highly  selected,  averaging  well  above  the  80th  percentile  on  their  admission 
tests,  yet  the  differences  in  time  they  require  to  learn  a  modestly  exotic  programming  language  remain  large. 

The  differences  in  the  speed  with  which  different  students  reach  given  objectives  may  be  due,  initially, 
to  ability,  but  this  effect  is  very  quickly  overtaken  by  prior  knowledge  as  a  determinant  of  pace 
(Tobias,  1989).  This  is  likely  to  be  particularly  true  of  students  in  military  education  and  training  who  bring  a 
wide  variety  of  backgrounds  and  life  experiences  to  the  classroom. 

The  challenge  this  diversity  presents  to  classroom  instructors  is  daunting.  How  can  they  ensure  that  every 
student  has  enough  time  to  reach  given  instructional  objectives?  At  the  same  time,  how  can  they  allow  those 
students  who  are  ready  to  surge  ahead?  The  answer,  of  course,  despite  heroic  efforts  to  the  contrary,  is  that 
they  cannot.  Most  classrooms  contain  many  students  who,  at  one  end  of  the  spectrum  are  bored  and,  at  the 
other  end,  are  overwhelmed  and  lost.  Technology  allows  us  to  alleviate  this  difficulty  by  adjusting  the  pace  of 
instruction  to  the  needs  and  abilities  of  individual  students.  They  can  proceed  as  rapidly  or  as  slowly  as 
needed.  They  can  easily  skip  what  they  already  know  or  have  mastered  and  concentrate  on  what  they  have  yet 
to  learn. 

Do  these  savings  in  time  come  as  the  expense  of  instructional  effectiveness?  The  data  suggest  the  opposite 
-  that  the  use  of  technology  both  decreases  instruction  time  and  increases  instruction  effectiveness. 
Noja’s  1987  findings  are  not  uncommon.  In  comparing  conventional  instruction  in  electronics  with 
technology-based  instruction  for  Italian  Air  Force  technicians,  he  found  a  reduction  in  training  time  of 
3  weeks  (from  8  to  5  weeks),  equivalent  student  achievement  for  electronic  theory,  and  substantial 
improvements  in  student  achievement  for  electronic  applications. 

One  study  does  not  provide  final  answers,  but  many  studies  can  be  aggregated  to  suggest  conclusions. 
This  aggregation  is  usually  done  using  “meta-analysis”  (analysis  of  analyses)  with  an  estimation  of  effect 
sizes.  Roughly,  effect  sizes  are  normalized  measures  of  standard  deviations  found  by  subtracting  the  mean 
from  one  collection  of  results  (e.g.,  a  control  group)  from  the  mean  of  another  (e.g.,  an  experimental  group) 
and  dividing  the  resulting  difference  by  an  estimate  of  their  common  standard  deviation.  Because  they  are 
normalized,  effective  sizes  can  be  averaged  to  give  an  overall  estimate  of  effect  from  many  separate  studies 
undertaken  to  investigate  the  same  phenomenon  or  treatment.  Figure  3  shows  effect  sizes  from  several 
collections  of  studies  that  compared  conventional  instruction  with  three  types  of  technology-based  instruction. 
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Figure  3:  Some  Effect  Sizes  for  Studies  Comparing  Technology-Based 
Instruction  with  more  Conventional  Approaches. 


In  Figure  3  ‘Computer-based  instruetion’  summarizes  results  from  233  studies  that  involved  straightforward 
applieation  of  eomputer  presentations  that  used  text,  graphies,  and  some  animation  -  as  well  as  some  degree 
of  individualized  interaetion.  The  effeet  size  of  0.39  standard  deviations  suggests,  roughly,  an  improvement  of 
50th  pereentile  students  to  the  performanee  levels  of  65th  pereentile  students. 

‘Interaetive  multimedia  instruetion’  involves  more  elaborate  interaetions  adding  more  audio,  more  extensive 
animation,  and  (espeeially)  video  elips.  The  added  eost  of  these  eapabilities  may  be  compensated  for 
by  greater  achievement  -  an  average  effect  size  of  0.50  standard  deviations  compared  with  an  effect  size  of 
0.39  for  typical  computer-based  instruction.  An  effect  size  of  0.50  for  interactive  multimedia  instruction 
suggests  an  improvement  of  50th  percentile  students  to  the  69th  percentile  of  performance. 

‘Intelligent  tutoring  systems’  involve  a  capability  that  has  been  developing  since  the  late  1960s 
(Carbonell,  1970),  but  has  only  recently  been  expanding  into  general  use.  In  this  approach  an  attempt  is  made 
to  directly  mimic  the  one-on-one  dialogue  that  occurs  in  tutorial  interactions.  The  key  component  is  that 
computer  presentations  and  responses  are  generated  in  real-time,  on  demand  and  as  needed  or  requested  by 
learners.  Mixed  initiative  dialogue  is  supported  in  which  either  the  computer  or  learner  can  ask  or  answer 
open-ended  questions.  These  interactions  are  generated  as  required.  Instructional  designers  do  not  need  to 
anticipate  and  pre-store  them.  This  approach  is  computationally  more  sophisticated  and  it  is  more  expensive  to 
produce  than  standard  computer-based  instruction.  However  its  costs  may  be  justified  by  the  increase  in 
average  effect  size  to  0.84  standard  deviations,  which  suggests,  roughly,  an  improvement  from  50th  to  80th 
percentile  performance. 

Some  later  intelligent  tutoring  systems  (Gott,  Kane,  and  Lesgold,  1995)  were  considered  just  to  see  how  far 
we  are  getting  with  this  approach.  The  average  effect  size  of  1.05  standard  deviations  for  these  recent 
applications  is  promising.  It  represents,  roughly,  an  improvement  of  the  performance  of  50th  percentile 
students  to  85th  percentile  performance. 
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The  more  extensive  tailoring  of  instruetion  to  the  needs  of  individual  students  that  ean  be  obtained  through  the 
use  of  generative,  intelligent  tutoring  systems  ean  only  be  expeeted  to  inerease.  Sueh  systems  may  raise  the 
bar  for  the  ultimate  effectiveness  of  technology-based  instruction.  They  may  make  available  far  greater 
efficiencies  than  we  can  now  obtain  from  other  approaches,  but  they  require  instructional  and  computational 
capabilities  that  today  are  only  partially  in  hand.  They  will  need  a  full  range  of  subject  matter  expertise  and 
sufficient  insight  to  present  it  to  students.  A  mathematical  model  of  a  system  will  help,  but  it  will  not  suffice. 
They  will  have  to  “know”  what  they  themselves  know  and  what  the  user  does  not.  They  will  need  an 
unobtrusive  assessment  capability  that  allows  them  to  dynamically  model  what  students  have  learned, 
have  not  yet  learned,  and  have  misunderstood.  Very  probably  they  will  need  to  communicate  with  their 
human  users  through  natural  (human)  language,  which  implies  not  just  speech-fragment  recognition, 
but  comprehensive  language  and  discourse  understanding.  We  will  need  the  intelligent,  if  not  spiritual 
machines,  Kurzweil  has  said  we  can  expect  in  20  years  -  or  so. 

Human  Instructors  in  the  Loop 

In  the  interim,  and  doubtless  after  that,  we  will  need  human  instructors  in  the  loop.  How  best  to  use  them  is  a 
perennial  issue  in  the  design  and  implementation  of  technology  based  instruction.  Finding  the  right  balance  is 
important.  Noja  has  attributed  much  of  his  considerable  success  in  preparing  technicians  for  Italian  Air  Force 
service  to  achieving  a  reasonable  balance  of  effort  between  technology  and  human  instructors  (1991). 

In  the  United  States,  as  in  perhaps  many  countries,  the  loss  (through  retirement,  vocational  bum-out, 
and  similar  factors)  of  military  experts  in  critical  areas  has  combined  with  drastic  reductions  in  live  training 
experiences  (due  to  costs,  range  and  environmental  restrictions,  greater  reach  of  weapons,  etc.)  has  increased 
the  need  to  make  best  use  of  remaining  military  expertise.  One  significant  way  to  do  this  is  to  support  experts 
who  are  performing  instmction  -  namely,  by  providing  technology-based  displays  that  are  intimately  linked 
to  the  subject  matter  being  presented  and  the  enabling  knowledge  and  skills  that  students  need  to  master  it. 
The  Interactive  Multi-sensor  Analysis  Training  (IMAT)  system  developed  by  the  US  Navy  to  support  training 
for  antisubmarine  warfare  sensor  operators  is  an  example  of  such  a  capability. 

IMAT  provides  training  in  sensor  deployment,  adversary  detection,  and  undersea  warfare  tactical  battlespace 
“sensemaking”  (e.g.,  environmental  analysis,  sensor  selection  and  placement,  search  rate  and  threat  detection, 
multi-sensor  crew  coordination,  and  multi-sensor  information  integration)  to  compensate  for  the  diminishing 
number  of  opportunities  to  develop  and  sustain  these  skills  in  formal  training,  on-the-job  training,  and  fleet 
exercises.  IMAT  integrates  models  of  the  physical  phenomena  with  innovative  visualization  techniques  in 
order  to  demonstrate  relationships  among  threats  (submarines  and  their  weaponry),  the  undersea  environment, 
and  anti-submarine  systems.  It  combines  analytic  and  instructional  design  technologies  with  advanced 
computer-based  graphics  to  promote  rapid  acquisition  of  the  cognitive  visualization  capabilities  sailors  need  to 
understand  structural  and  spatial  interrelationships  among  sensors,  platforms,  and  submarine  systems. 

The  combination  of  instructor-led,  IMAT-supported  training  has  not  only  raised  performance  against 
established  instructional  objectives,  but  it  has  also  raised  the  level  of  objectives  set  for  training.  It  helped 
sailors  achieve  levels  of  competence  that  simply  were  not  available  without  it.  Notably,  IMAT  fulfills  most  of 
the  training  capabilities  suggested  by  Figure  1.  It  allows  training  to  be  ported  from  basic  to  complex 
operational  environments.  It  allows  training  environments  using  human  instructors  and  advanced  technology 
together  to  achieve  significantly  more  high-level  (transferable,  abstract,  problem-solving)  objectives  than  were 
attainable  using  traditional  training  approaches  and  does  so  at  easily  affordable  costs.  It  supports  training  in 
familiar,  relatively  simple  environments  while  being  scaleable  to  environments  characteristic  of  “live” 
military  operations.  Instructor-led,  IMAT  supported  training  allows  students  to  accomplish  far  more  than 
either  instructors  or  technology  can  achieve,  working  separately. 


6-8 


RTO-MP-088 


Computer-Based  Aids  for  Learning,  Job  Performance,  and  Decision 
Making  in  Military  Applications:  Emergent  Technology  and  Challenges 


Performance  Aiding  Applications 

Problem  solving  is  required  when  an  individual  or  a  group  must  achieve  a  goal  but  are  uncertain  how  to  do  so 
(Baker  and  Mayer,  1999;  Mayer  and  Wittrock,  1996).  It  requires  ingenuity  and  creativity  to  manipulate  and 
transform  the  knowledge  and  skills  that  problem  solvers  possess  into  paths  of  action  leading  to  the  goal.  It  is  a 
necessary  and  integral  component  of  human  performance  in  all  sectors  and  it  is  a  critical  component  in  every 
military  operation.  Difficulty  in  making  problem  solving  decisions  is  indicated  by  the  frequency  with  which 
we  are  confronted  with  too  much  data,  too  many  options,  and  unknown  levels  of  risk.  These  matters  have  been 
the  object  of  systematic  study  by  researchers  both  past  (James,  1890/1950)  and  present  (Edwards  and  Fasolo, 
2001). 

Given  the  complexity  of  real  world  decision-making  and  the  range,  both  descriptive  and  prescriptive,  of  its 
theoretic  underpinnings,  it  does  not  seem  unreasonable  to  seek  assistance  from  technology.  Such  applications 
involve  tightly  integrated  interactions  between  humans  and  automated  systems.  They  raise  some  of  the  most 
interesting  issues  for  research  and  development  in  determining  the  role  of  humans  in  the  use  of  intelligent  and 
automated  systems. 

There  are  not  enough  studies  of  the  effectiveness  and  cost-benefits  of  these  applications  to  permit  a  meta- 
analytic  assessment  of  their  impact.  However,  systematic  and  comprehensive  assessment  of  technology-based 
performance  aiding  can  be  found  in  the  collection  of  studies  covering  the  Integrated  Maintenance  Information 
System  (IMIS),  an  automated,  hand-held,  flight- line  avionics  maintenance  aid.  Tomasetti,  et  al.  (1993) 
documented  a  thorough  cost  analysis  of  IMIS.  Thomas  later  (1995)  reported  results  from  an  empirical 
investigation  of  IMIS  effectiveness.  Teitelbaum  and  Orlansky  (1996)  summarized  results  from  both  these 
studies,  combined  them  into  a  more  complete  cost-effectiveness  assessment,  and  discussed  the  implications  of 
these  findings. 

Thomas  (1995)  compared  the  performance  of  12  Avionics  Specialists  and  12  Airplane  General  (APG) 
Technicians  on  12  fault  isolation  problems  concerning  three  F-16  avionics  subsystems  -  fire  control  radar, 
heads-up  display,  and  inertial  navigation.  Within  each  of  the  two  groups  of  subjects,  six  of  the  fault  isolation 
problems  were  performed  using  paper-based  Task  Orders  (TOs  -  Air  Force  technical  manuals)  and  six  were 
performed  using  IMIS.  Training  for  APG  Technicians  includes  all  aspects  of  aircraft  maintenance,  only  a 
small  portion  of  which  concerns  avionics.  In  contrast.  Avionics  Specialists  receive  16  weeks  of  specialized 
training  in  avionics  maintenance.  Results  of  the  study  are  shown  in  Table  2. 


Table  2:  Maintenance  Performance  of  12  Air  Force  Avionics  Specialists  and 
12  General  (APG)  Technicians  Using  Technical  Orders  (TOs)  and  IMIS 


Technicians/ 

Performers 

Correct 

Solutions 

(Percent) 

Time  to 
Solution 
(Minutes) 

Average 
Number  of 
Parts  Used 

Time  to 
Order  Parts 
(Minutes) 

TOs 

IMIS 

TOs 

IMIS 

TOs 

IMIS 

TOs 

IMIS 

Avionics  Specialists 

81.9 

100.0 

149.3 

123.6 

8.7 

6.4 

19.4 

1.2 

APG  Technicians 

69.4 

98.6 

175.8 

124.0 

8.3 

5.3 

25.3 

1.5 

Observations  that  might  be  made  from  these  results  include  the  following: 

(a)  Avionics  Specialists  using  Task  Orders  compared  with  those  using  IMIS.  The  Avionics  Specialists 
using  IMIS  found  more  correct  solutions  in  less  time,  used  fewer  parts  to  do  so,  and  took  less  time  to 
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order  them.  All  these  results  were  statistically  significant.  The  number  of  parts  required  may  deserve 
brief  comment  -  savings  in  spare  parts  inventory  and  transportation  were  by  far  the  largest  factors  in 
the  Tomasetti,  et  al.  (1993)  analysis  of  costs.  The  parts  reduction  exerted  considerable  leverage  on 
the  overall  cost  savings  reported  by  Teitlebaum  and  Orlansky  (1996).  The  reduction  in  time  to  order 
parts  is  to  be  expected  because  IMIS  automates  much  of  this  process.  Notably,  the  time  taken  by 
technicians  to  complete  the  paperwork  in  the  absence  of  IMIS  could  be  used  elsewhere  with 
substantial  productivity  gains  and  cost  savings  if  IMIS,  or  a  similar  capability,  performs  these 
paperwork  chores. 

(b)  APG  Technicians  using  Task  Orders  compared  with  those  using  IMIS.  Thomas  found  similar 
results  in  these  comparisons.  APG  Technicians  using  IMIS  performed  more  correct  solutions  in  less 
time,  used  fewer  parts  to  do  so,  and  took  less  time  to  order  them.  As  with  Avionics  Specialists, 
all  these  results  were  statistically  significant. 

(c)  APG  Technicians  using  IMIS  compared  with  Avionics  Specialists  using  Task  Orders. 
APG  Technicians  using  IMIS  found  more  correct  solutions  in  less  time,  used  fewer  parts  to  do  so, 
and  took  less  time  to  order  them  than  did  Avionics  Specialists  using  paper-based  Task  Orders. 
All  these  results  were  statistically  significant.  This  result  suggests  that  it  is  feasible  and  desirable  to 
replace  some  of  the  extra  training  required  by  specialists  with  on-the-job,  just-in-time  decision  aids, 
such  as  IMIS,  supplied  to  non-specialists. 

(d)  APG  Technicians  using  IMIS  compared  with  Avionics  Specialists  using  IMIS.  In  these 
comparisons,  APG  Technicians  performed  about  as  well  as  Avionics  specialists,  and  even  slightly 
better  in  the  number  of  parts  used.  None  of  these  comparisons  were  statistically  significant  and  none 
appear  to  be  practically  significant.  These  results  again  suggest  the  feasibility  of  replacing  some 
number  of  specialists,  who  require  greater  training  costs,  with  general  technicians  supplied  with 
on-the-job,  just-in-time  decision  aids.  They  also  suggest  the  desirability  of  doing  so,  because  of  the 
greater  costs  to  train  the  specialists  -  even  though  the  resulting  performance  on  the  job,  where  it 
counts,  is  the  same  in  both  cases. 

The  promise  suggested  by  these  results  could  well  vanish  if  the  costs  to  provide  the  decision  aid  (IMIS) 
exceed  the  costs  they  otherwise  save.  Enter  the  costs  and  benefits  analysis  by  Tomasetti  et  al.  (1993) 
combined  with  the  empirical  results  reported  by  Thomas  (1995)  -  using  these  two  sources  of  data,  Teitelbaum 
and  Orlansky  (1996)  were  able  to  estimate  reductions  in  depot- level  maintenance,  organizational-level 
maintenance,  and  the  maintenance  and  transportation  of  inventories  of  spare  parts.  Teitelbaum  and  Orlansky 
estimated  annual  savings  from  the  use  of  IMIS  of  approximately  $38  million  for  the  full  Air  Force  fleet 
of  about  1,700  F-16s.  Their  analysis  also  considered  the  costs  to  develop  and  maintain  IMIS.  Assuming  an 
8-year  useful  life  for  IMIS,  they  arrived  at  a  figure  of  about  $18  million  per  year  to  maintain  IMIS 
(including  its  databases)  and  to  amortize  its  development  costs.  The  result  is  about  $20  million  per  year  in  net 
savings. 

This  figure  of  $20  million  is  conservative.  Focusing  only  on  cost  may  underestimate  the  total  “value”  of  this 
technology.  It  does  not  include:  (a)  savings  that  would  result  from  a  reduction  in  Air  Force  requirements  to 
recruit  and  train  specialized  personnel  such  as  the  Avionics  Specialists  in  Thomas’s  study;  (b)  savings  in 
training  that  would  accrue  from  the  use  of  IMIS  as  both  a  decision  aid  and  a  training  device;  (c)  savings  in  the 
costs  to  print,  distribute,  and,  especially,  update  paper  technical  manuals;  and  (d)  savings  (of  about  50  percent) 
in  time  to  debrief  pilots  about  maintenance  problems.  Most  important,  these  benefits  do  not  include  those 
arising  from  increased  sortie  rates  and  enhanced  operational  readiness  and  effectiveness  resulting  from  the 
substantially  improved  problem  solving  competencies  of  maintenance  personnel. 
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As  with  training  systems,  performance  aiding  will  benefit  from  what  appears  to  be  the  inexorable  march  of 
technology.  But  significant  technology  challenges  remain.  Performance  aiding  systems  will  require 
interactive,  advising,  decision  aiding,  and  computational  capabilities  that  today  are  only  partially  in  hand. 
Just  like  training  systems,  they  will  need  a  full  range  of  subject  matter  expertise  and  sufficient  insight  to 
present  it  to  users.  They  will  have  to  “know”  what  they  themselves  know  and  what  the  individual  user  does 
not  so  they  provide  assistance  in  ways  that  the  user  is  prepared  to  understand  and  act  on.  In  order  to  do  this, 
they  will  need  an  unobtrusive  assessment  capability  that  allows  them  to  dynamically  model  what  users  know, 
do  not  know,  and  misunderstand.  Very  probably  they  will  need  to  communicate  with  their  human  users  using 
natural  (human)  language  involving  comprehensive  language  and  discourse  understanding.  They  too  will  need 
the  “intelligent”  capabilities  for  interacting  with  humans  that  is  foreseen  by  Kurzwiel  (and  others)  and  that 
researchers  developing  intelligent  tutoring  systems  are  creating.  These  systems  are  discussed  in  the  next 
section. 

Is  It  Worth  It?  A  Summary 

Given  the  weight  of  data  presented  here  and  elsewhere,  it  seems  reasonable  to  accelerate,  research, 
development,  and  implementation  of  computer  technology  in  military  education,  training,  and  performance 
aiding.  Resources,  however,  for  such  a  pursuit  are  finite  and,  most  likely,  cannot  support  all  that  remains  to  be 
done.  Thus,  some  consideration  of  technology  opportunities  and  priorities  driven  by  operational  requirements 
seems  in  order.  We  should  decide  where  we  want  to  go  before  trying  to  get  there. 


WHERE  DO  WE  WANT  TO  GO? 

One  prominent  and  promising  vision  for  the  future  is  captured  by  the  notion  of  asynchronous,  continuous 
learning,  which  is  education,  training,  and  performance  aiding  that  is  available  anytime,  anywhere  to 
whomever  needs  it.  Such  a  vision  capitalizes  on  the  development  of  the  world-wide  web,  or  whatever  the 
global  information  grid  of  the  future  will  be,  generative  computer-based  instruction,  unobtrusive  assessment 
of  users’  intentions  and  needs,  intelligent  tutoring  systems,  natural  language  understanding,  the  development 
of  educational  objects,  and,  of  course,  the  continuing  advancements  in  computation  and  computer  technology. 
This  vision  is  roughly  captured  by  Figure  4. 


RTO-MP-088 


6-11 


Computer-Based  Aids  for  Learning,  Job  Performance,  and  Decision 
Making  in  Miiitary  Appiications:  Emergent  Technology  and  Challenges 


ORGANIZATION 


Shareable  courseware  Assembled  In  To  provide  learning 

objects  from  across  the  real-time,  on-  and  assistance 

World  Wide  Web  demand  anytime,  anywhere 


Figure  4:  Asynchronous,  Continuous  Learning:  A  Vision  for  the  Future. 


This  vision  anticipates  a  future  in  which  everyone  may  have  an  electronic  personal  learning  associate. 
This  device  will  assemble  learning  or  decision-aiding  presentations  on  demand  and  in  real-time  -  any  time, 
anywhere.  The  presentations  will  be  exactly  tailored  to  the  needs,  capabilities,  intentions,  and  learning  state  of 
each  individual  or  group  (e.g.,  crew,  team,  or  staff).  Communication  with  the  device  will  be  based  on  natural 
language  dialogue  initiated  either  by  the  device  or  by  its  users.  The  device  will  small  enough  to  be  carried  in  a 
shirt  pocket,  or  it  will  be  wearable.  It  will  be  used  by  individuals  learning  by  themselves,  in  groups,  or  in 
classrooms.  It  will,  of  course,  be  wireless. 

Most  of  the  technology  needed  to  build  such  a  device  exists  now.  Although  we  cannot  yet  fit  it  into  a  shirt 
pocket,  creative  engineering  should  take  care  of  that.  What  is  especially  needed  for  instruction  and  decision- 
aiding  is  content  in  the  form  of  instructional  objects,  which  we  are  calling  shareable  instructional  objects. 
These  objects,  shown  in  the  cloud  on  the  left  side  of  Figure  4,  must  be  readily  accessible  across  the  World 
Wide  Web  -  or  whatever  form  our  global  information  network  takes  in  the  future. 

Once  these  objects  exist,  they  must  be  identified,  selected,  and  assembled  in  real-time,  on  demand  and  then 
handed  to  the  personal  learning  associates,  which  provide  the  instruction  or  decision  aiding.  This  work  of 
identifying,  selecting,  and  assembling  objects  is  the  job  of  the  server,  the  box  in  the  middle  of  Figure  4. 
By  importing  ‘logic’  or  instructional  strategy  objects,  the  server,  which  today  might  be  called  a  learning 
management  system,  can  acquire  the  capabilities  of  an  intelligent  tutoring  system.  This  vision  is  keyed  to  the 
development  and  implementation  of  intelligent  instructional  systems.  These  systems  must  be  scaleable  and 
adaptive  to  military  personnel  ‘occupational  specialties’,  the  variety  of  instruction  venues,  and  the  operational 
domain(s)  addressed  by  the  application  (mapped  with  notional  numbered  examples  in  Figure  5). 


6-12 


RTO-MP-088 


Computer-Based  Aids  for  Learning,  Job  Performance,  and  Decision 
Making  in  Military  Applications:  Emergent  Technology  and  Challenges 


Figure  5:  Dimensionality  of  Instructional  Systems. 


Figure  5  represents  the  ehallenge  to  designer  and  developers  of  military  instruetional  systems  along  three 
dimensions. 

•  The  “Student”  axis  eoncerns  the  types  of  student  eoupled  with  the  roles  and  responsibilities  of 
the  student.  These  roles  and  responsibilities  suggest  targeted  levels  of  expertise,  abstraction, 
and  objectives.  This  axis  points  to  the  need  for  dynamic  adaptability  of  the  instructional  system. 

•  The  “Instructional  Venue”  axis  reflects  where  the  instruction  will  occur  -  in  the  classroom,  the  work 
environment,  duty  station,  garrison,  home,  or  some  combination  of  these.  This  axis  points  to  the  need 
for  curriculum  design  and  the  instructional  infrastructure  flexibility  and  its  required  characteristics. 

•  The  “Student  Location”  axis  identifies  where  the  knowledge  gained  will  be  used  -  in  locations 
ranging  from  garrison  and  depots,  to  field  combat  command  centers,  to  the  battlefield.  It  identifies 
both  the  application  environment  for  the  knowledge  and  skills  gained  and  their  criticality. 

The  figure  suggests  the  kinds  of  intelligence  that  an  automated  instructional  system  must  possess  to  operate 
effectively  on  behalf  of  its  human  user.  It  also  suggests  dimensions  for  validating  instructional  and 
performance  aiding  systems  to  ensure  that  the  instruction  and  information  they  convey  can  be  trusted. 
Research  and  development  along  these  dimensions  will  directly  impact  what  are  commonly  called  intelligent 
tutoring  systems. 
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Intelligent  Tutoring  Systems  (ITS) 

It  may  be  best  to  note  the  features  ‘ordinary’  computer-based  instruction  provides:  it  can  (a)  accommodate 
individual  students’  rate  of  progress,  allowing  as  much  or  as  little  time  as  each  separate  student  needs  to  reach 
instructional  objectives;  (b)  adjust  the  sequence  of  instructional  content  to  each  student’s  needs;  (c)  adjust  the 
content  itself  -  different  students  can  receive  different  content  depending  on  what  they  have  mastered  and 
what  they  have  yet  to  learn;  (d)  make  the  instruction  as  easy  or  as  difficult  as  necessary;  and,  (e)  adjust  to  the 
learning  style  (e.g.,  verbal  versus  visual)  that  is  most  appropriate  for  each  student.  These  capabilities  have 
been  available  and  used  in  computer-based  instruction  from  its  inception  in  the  1950s  (Fletcher  &  Rockway, 
1986).  Those  who  promote  systems  with  these  features,  touting  them  as  indicators  of  newly  developed 
‘intelligent’  capabilities,  may  be  missing  some  history.  They  are  using  the  term  ‘intelligent’  in  ways  that  differ 
from  the  historical  objectives  of  ITS  -  objectives  that  have  been  pursued  since  the  late  1960s. 

What  are  these  objectives?  What  is  left  for  ITS  to  provide?  What  can  we  get  from  them  that  is  not  otherwise 
available?  Two  functionalities  deserve  mention: 

•  The  ability  to  allow  either  the  computer  or  the  student  to  ask  open-ended  questions  and  initiate 
instructional,  “mixed-initiative”  dialogue  as  needed  or  desired. 

•  The  ability  to  generate  instructional  material  and  interactions  on  demand  rather  than  require 
developers  to  foresee  and  pre-store  all  such  materials  and  interactions  needed  to  meet  all  possible 
eventualities. 

The  first  of  these  functionalities  requires  the  ITS  to  understand  and  participate  in  mixed-initiative  interactions 
with  the  student.  It  requires  mutual  understanding  of  a  language  for  information  retrieval,  decision-aiding, 
and  instruction  that  is  shared  by  both  the  ITS  and  the  student/user.  Natural  language  has  been  a  frequent 
choice  for  this  capability,  but  the  language  of  mathematics,  mathematical  logic,  and  electronics  have  also  been 
used  (e.g.,  Suppes,  1981;  Sleeman  and  Brown,  1982;  Psotka,  Massey,  and  Mutter,  1988;  and  Farr  and  Psotka, 
1992).  Whatever  form  it  takes,  mixed-initiative  dialogue  in  which  either  the  student  or  the  instructor  can 
initiate  interactions  is  a  key  feature  of  one-on-one  tutorial  instruction  (e.g.,  Graesser,  Person,  &  Magliano, 
1995).  Such  a  capability  has  long  been  a  goal  of  intelligent  tutoring  systems  (Carbonell,  1970). 

The  second  functionality  requires  ITS  to  devise  on  demand  -  not  retrieve  from  storage  -  interactions  and 
presentations  for  individual  students.  This  capability  involves  more  than  generating  elements  to  fill  in  blanks 
in  a  template.  It  means  generating  interactions  and  presentations  from  information  primitives  using  an 
“instructional  grammar”  that  is  analogous  to  the  deep  structure  grammar  of  the  transformational-generative 
linguists  of  a  generation  ago.  This  functionality  harkens  back  to  the  roots  of  ITS  development  as  (again) 
can  be  seen  in  the  volumes  edited  by  Suppes  (1981),  Sleeman  and  Brown  (1982),  Psotka,  Massey,  and  Mutter 
(1982),  and  Farr  and  Psotka  (1992). 

Motivations  for  both  these  functionalities  can  be  found  in  basic  research  into  human  learning,  memory, 
perception,  and  cognition.  Findings  from  this  research  have  led  us  to  view  all  cognitive  processes  as 
constructive  and  regenerative.  They  have  caused  general  theories  of  perception  and  learning  to  evolve  from 
the  fairly  strict  logical  positivism  of  behavioral  psychology,  which  emphasized  the  study  of  directly 
observable  and  directly  measurable  actions,  to  greater  consideration  of  internal,  less  observable  processes  that 
are  assumed  to  mediate  and  enable  human  learning  -  and  to  produce  the  directly  observable  behavior  that  is 
the  subject  of  behaviorist  approaches.  The  keynote  of  these  conceptions  of  cognition  may  have  been  struck  by 
Ulric  Neisser,  who  stated,  “The  central  assertion  is  that  seeing,  hearing,  and  remembering  are  all  acts 
of  construction,  which  may  make  more  or  less  use  of  stimulus  information  depending  on  circumstances” 
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(1967,  p.lO).  But  these  ideas  have  been  part  of  the  fabrie  of  seientific  psychology  since  its  inception 
(e.g.,  James,  1890/1950). 

This  point  of  view  suggests  that  the  generative  capability  sought  by  the  vision  presented  by  Figure  4  is  not 
something  merely  nice  to  have,  but  is  essential  if  we  are  to  advance  beyond  the  constraints  of  the  prescribed 
branching,  programmed  learning,  and  the  ad-hoc  principles  currently  used  to  design  technology-based 
instruction.  A  generative  approach  may  be  essential  if  we  are  to  deal  successfully  with  the  immensity,  extent, 
and  variability  of  human  cognition. 

The  key  defining  characteristic  of  intelligent  tutoring  systems,  then,  is  not  application  of  computer  techniques 
from  artificial  intelligence  or  knowledge  representation,  or  the  specification  of  sharable  instructional  objects, 
important  as  these  may  be.  It  is  rather  the  functional  capability  to  generate  in  real-time  and  on-demand 
instructional  interactions  that  are  tailored  to  student  requests  and/or  needs.  This  generative  capability 
motivated  the  U.S.  Department  of  Defense  to  invest  in  the  development  of  these  systems  in  the  first  place 
(Fletcher  &  Rockway,  1986).  At  that  time  the  motivation  was  to  reduce  or  eliminate  the  high  costs  of 
foreseeing  or  predicting  all  possibly  needed  materials  and  interactions,  programming  them,  and  pre-storing 
them  into  computer-based  instruction.  Today,  this  motivation  remains  fundamental  to  research  and 
development  in  intelligent  systems  intended  to  augment  the  learning  and  performance  of  human  beings. 

An  Example  of  an  Intelligent  Tutoring  System  at  Work 

An  example  of  these  capabilities  at  work  may  be  found  in  the  student- computer,  mixed-initiative  dialogues 
supported  by  SOPHIE  (Sophisticated  Instructional  Environment)  -  an  early  and  seminal  ITS  (Brown,  Burton, 
&  DeKleer,  1982)  that  would  fit  in  Figure  5  at  the  lower-ffont-left.  SOPHIE  used  a  general-purpose  electronic 
simulator  to  represent  an  electronic  power  supply  and  create  a  reactive,  tutorial  environment  for  training  in 
electronic  troubleshooting.  The  underlying  simulator  allowed  faults  introduced  into  the  system  by  a  (human) 
instructor  or  by  SOPHIE  itself  to  be  propagated  throughout  the  (simulated)  power  supply  in  a  realistic  fashion. 
SOPHIE  employed  a  semantic  network  of  information  about  the  power  supply,  one  of  the  most  sophisticated 
natural  language  understanders  ever  built  into  ITS,  and  an  articulate,  expert  troubleshooting  capability  that 
could  not  only  solve  arbitrary  faults  introduced  by  students  or  instructors  but  also  explain  to  the  student  how  it 
solved  them. 

SOPHIE’S  reactive  environment  responded  in  a  realistic  fashion  to  the  actions  and  decisions  of  a  student 
engaged  in  practical  problem  solving.  It  employed  strategies  that  shadowed  those  used  by  the  student  to  make 
inferences  about  the  circuit  based  on  the  student’s  measurements.  SOPHIE  could  thereby  determine  which 
deductions  the  student  was  using,  construct  a  model  of  his/her  state  of  knowledge,  and  use  this  model  to 
generate  explanations  in  terms  that  the  student  was  prepared  to  understand.  SOPHIE  highlighted  the  need  for 
intelligent  training  systems  not  only  to  solve  arbitrary  problems,  but  to  do  so  in  a  way  that  can  be  explained  to 
students  and  be  understood  by  them.  It  pioneered  capabilities  currently  lacking  in  many  information 
acquisition  and  retrieval  systems. 

The  following  is  a  student-computer  dialogue  adapted  from  SOPHIE: 

Student:  What  is  the  base  emitter  voltage  of  the  voltage  limiting  transistor? 

SOPHIE:  The  base  emitter  voltage  of  Q5  is  -.037  volts. 

The  first  student-SOPHIE  interaction  in  this  example  is  a  straightforward  question  and  answer.  SOPHIE  has 
introduced  a  fault  into  the  simulated  power  supply  and  the  student  is  trying  to  locate  and  repair  it. 
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Student:  What  about  for  the  current  limiting  transistor? 

SOPHIE:  The  base  emitter  voltage  of  Q8  is  .349  volts. 

The  second  student-SOPHIE  interaction  is  a  little  less  straightforward  because,  in  addition  to  simulating  the 
faulted  power  supply,  SOPHIE  must  be  able  to  parse  the  dialogue  sufficiently  to  know  that  the  student  is 
asking  for  the  base  emitter  voltage  of  transistor  Q8.  The  meaning  of  the  student’s  question  depends  heavily  on 
the  context  established  by  his  prior  question. 

Student:  What  should  it  be? 

SOPHIE:  In  a  working  circuit,  the  base  emitter  voltage  of  Q8  is  .593  volts. 

The  third  student-SOPHIE  interaction  continues  its  dependence  on  SOPHIE’s  understanding  of  context  and 
additionally  requires  SOPHIE  to  extract  the  answer  by  simulating  both  the  faulted  power  supply  and  one  that 
is  operating  correctly. 

Student:  Replace  Q8. 

SOPHIE:  I  am  going  to  ask  you  some  questions  about  how  Q8  is  faulted. 

Are  any  junctions  shorted? 

The  fourth  student-SOPHIE  interaction  indicates  a  clear  step  beyond  what  Brown  et  al.  described  as  a 
knowledgeable  system  to  what  they  considered  to  be  an  intelligent  system.  SOPHIE  has  progressed  from  a 
knowledgeable  parsing  of  its  dialogue  with  the  student  and  simulation  of  various  states  of  the  power  supply  to 
a  system  exercising  tutorial  intelligence.  It  shadowed  the  student’s  solution  path,  modeled  the  student’s 
troubleshooting  hypotheses,  determined  that  they  are  incorrect,  elected  to  capture  the  dialogue  initiative  back 
from  the  student,  and  is  undertaking  a  series  of  tutorial  interactions  intended  to  lead  the  student  back  to  a  more 
correct  approach  to  the  problem. 

It  is  difficult  to  imagine  any  practical  way  to  achieve  this  level  of  functionality  without  the  generative 
capability  and  mixed- initiative  dialogue  that  distinguishes  intelligent  tutoring  systems  from  other  forms  of 
computer-based  instruction.  Such  functionality  is  as  applicable  and  essential  to  performance  aiding  as  it  is  to 
instruction. 

To  this  point  we  have  described  a  vision  of  computer-based  instruction  in  general  training  or  education 
application  which  in  a  military  context  is  the  foundation  for  developing  domain  expertise  in  military 
specialties.  Can  this  vision  be  applied  to  the  adult  learning,  occupational  training  domain  that  is  the  foundation 
of  military  effectiveness? 

The  answer  should  be  “yes”,  but  that  yes,  as  suggested  earlier  in  this  paper,  is  limited  by  the  instructional, 
performance  aiding,  and  computational  capabilities  that  are  available  today.  We  have  speech  recognition, 
but  the  levels  of  language  and  discourse  understanding  we  need  remain  to  be  developed.  We  have  valid 
models  of  military  systems,  but  we  lack  the  “executive  functions”  that  will  allow  automated  instructional  and 
performance  aiding  systems  to  map  the  expertise  represented  by  these  models  onto  human  users  and  match 
their  presentations  to  individual  users’  needs.  We  have  authoring  systems,  but  they  need  additional 
development  to  reduce  to  an  acceptable  level  the  costs  and  time  required  to  produce  instructional  and 
performance  aiding  systems.  Above  all  we  lack  a  comprehensive  set  of  empirically-based  design  principles  to 
support  an  engineering  of  instructional  and  performance  aiding  systems  that  would  lead  reliably  to  the 
achievement  of  given  instructional  outcomes  and  substantially  reduce  the  time  to  produce  human  expertise 
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and  competence,  or  even,  as  IMAT  does,  produce  levels  of  expertise  in  military  personnel  that  currently  do 
not  exist.  What  is  needed  to  produce  such  capabilities  is  the  topic  of  the  next  section. 

Where  Do  We  Want  To  Go?  A  Summary 

In  brief,  the  vision  depicted  by  Figure  4  and  applied  in  the  dimensions  portrayed  by  Figure  1  and  Figure  5 
implies  a  number  of  key  goals,  among  which  are  the  following: 

•  The  product  of  this  research  and  development  must  be  capable  of  generating  presentations  on 
demand,  in  real  time. 

•  Content  analysis  (e.g.,  cognitive  task  analysis)  must  be  able  to  be  conducted  by  trainers  and  educators 
ubiquitously. 

•  Its  presentations  must  be  tailored  in  content,  sequence,  level  of  difficulty,  level  of  abstraction,  style, 
etc.  to  users’  intentions,  backgrounds,  and  needs.  Neither  student  nor  problem  solvers  will  benefit 
from  a  device  that  does  not  provide  information  they  are  capable  of  understanding.  Efficiency  in  both 
learning  and  performance  also  demands  such  tailoring  to  user  needs  and  capabilities. 

•  It  must  be  used  equally  well  to  aid  learning  or  performance. 

•  It  must  accommodate  mixed  initiative  dialogue  in  which  either  the  technology  or  the  user  can  initiate 
or  respond  to  inquiries  in  natural  language. 

•  It  will  benefit  greatly  from  a  supply  of  sharable  instructional  objects  readily  available  for  the 
generation  of  instructional  (or  decision  aiding)  presentations. 

These  goals  suggest  requirements  for  research  and  development  -  requirements  that  must  be  met  if  we  are  to 
get  from  here  (now)  to  there  (where  we  want  to  be). 


HOW  SHOULD  WE  GET  FROM  HERE  TO  THERE? 

There  remains  much  to  be  done  before  we  will  achieve  the  vision  for  education,  training,  and  performance 
aiding  presented  in  this  paper.  However,  accomplishment  of  some  research  and  development  goals  will  result 
in  products  of  immediate  utility  and  might  well  be  sought  in  the  short  term. 

The  foundation  for  these  recommendations  was  laid  by  a  US  Department  of  Defense  workshop  on  continuous 
learning  systems  held  in  October  1999.  It  was  motivated  in  large  part  by  the  recognition  that  military 
readiness  and  effectiveness  are  keyed  not  just  to  the  supply  of  systems  and  personnel,  which  are  commonly 
measured  in  US  assessments  of  readiness,  but  also  to  the  cognitive  capabilities  of  military  personnel  -  to  what 
might  be  described  as  their  cognitive  readiness  (Etter,  Foster,  and  Steele,  2000). 

Table  3  is  both  a  summary  and  an  elaboration  of  findings  from  this  workshop.  It  is  roughly  organized  into 
the  standard  system  engineering  categories  of  analysis,  design  and  development,  delivery  and  management, 
and  evaluation. 

The  table  lists  and  briefly  describes  potential  elements  of  a  research  agenda  for  the  development  of 
technology-based  aids  for  education,  training,  performance  aiding,  and  decision  making  in  military 
applications.  These  elements  do  not  comprise  a  formally  established  research  agenda,  but  they  suggest  the 
breadth  of  research  and  development  that  must  be  undertaken  to  capitalize  on  the  promise  offered  by  our  new 
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technologies  for  education,  training,  and  performance  aiding.  The  elements  suggest  opportunities  for 
cooperative,  international  development  undertaken  by  NATO/PfP  countries. 


Table  3:  Capabilities,  Goals  and  Research  and  Development  Requirements  for  the 
Development  of  Technology  in  Education,  Training  and  Performance  Aiding 


Capability 

Description/Goal 

Requirements/Gaps 

Instruction  Analysis: 
Cognitive  Theory 

Accurate  and  comprehensive 
representation  of  cognitive  skill 
and  cognitive  skill  development 
to  include  such  ‘higher  order’ 
capabilities  as  decision-making, 
problem-solving,  meta-cognition, 
pattern  recognition,  critical 
thinking,  situational  awareness, 
and  teamwork. 

New  concepts  and  theories  of  cognition 
based  on  new  measurement  capabilities, 
especially  those  available  from 
technology-based  assessment; 
Representation  of  expertise  in  complex, 
ill-structured  environments. 

Instruction  Analysis: 
Training  Analysis 

Efficient,  comprehensive,  and 
accurate  analysis  of  military 
needs  for  command,  job,  and 
task  performance. 

Automation  of  processes  for  performing 
front  end  analysis,  cognitive  analysis, 
and  cognitive  readiness  assessment; 
Comprehensive  and  accurate 
representation  of  individual  and  crew, 
team,  and  unit  expertise,  capabilities, 
and  performance;  Principles  for 
determining  relevant  task  characteristics 
for  instructional  design  and  selection 
of  instructional  strategies. 

Instruction  Analysis: 

Integrated  human 
performance  system 

Training  analysis,  design,  and 
selection  of  objectives  integrated 
with  the  full  range  of  alternatives 
used  to  ensure  human  competence 
in  military  operations. 

Comprehensive  principles  for  optimal, 
cost-effective,  and  balanced  adjustment 
of  trade-offs  between  all  elements 
(e.g.,  selection,  personnel  classification, 
training,  ergonomic  design,  job  design) 
used  to  ensure  the  presence  of  human 
competence  when  and  where  it  is  needed. 

Instruction  Analysis: 

T  echno  logy-based 

knowledge 

representation 

Verified  and  validated 
representations  of  expert 
solutions,  capabilities, 
and  behavior. 

Automated  capture  of  expertise; 
Self-generating,  self-modifying  data 
bases  built  from  cases  and  examples 
of  successful  problem  solving  and 
decision  making. 

Instruction  Analysis: 

Assessment  of 
learner  potential 

Ability  to  use  computer  technology 
to  develop  more  comprehensive 
and  accurate  models  of  human 
cognition  and  potential  -  ability  to 
identify  potential  ‘aces’  in  specific 
militarily  relevant  areas. 

New  forms  of  computer-administered 
assessment  items  using  the  full  display 
and  timing  capabilities  of  technology. 
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Capability 

Description/Goal 

Requirements/Gaps 

Instruction  Design 
and  Development: 

Instructional  Objects 

Wide  availability  of  accessible, 
portable,  reusable  instructional 
components  or  ‘objects’;  Fully 
developed  infrastructure  to 
support  their  use. 

Object  oriented  programming  for 
instruction,  models,  and  simulations; 
Standards  for  developing  sharable 
instructional  and  simulation  objects. 

Instruction  Design 
and  Development: 

Management  of 

Student  Progress 

Comprehensive  engineering  of 
instruction  that  links  instructional 
design  and  presentations  to  targeted 
instructional  outcomes  (e.g.,  speed 
of  response,  accuracy  of  response, 
retention  of  knowledge  and  skills, 
motivation  to  continue  learning). 

Principles  of  design  and  presentation 
that  can  be  automated  as  needed  to 
ensure  reliable  achievement  of 
instructional  outcomes  by  individuals, 
crews,  teams,  and  units. 

Instruction  Design 
and  Development: 

Cognitive  readiness 
development 

Development  of  a  comprehensive 
techniques  for  measuring  and 
enhancing  cognitive  readiness. 

Principles  for  the  development  of  higher 
level  cognitive  skills  such  as  creativity, 
intuition,  pattern  recognition,  transfer 
of  knowledge  and  skills,  problem 
solving,  and  situation  awareness. 

Instruction  Design 
and  Development: 

Human-behavior 

representation 

Verified  and  valid  representation 
of  individual  and  group  behavior 
for  virtual  team  members, 
adversaries,  friendly  forces, 
non-combatants,  and 
organizational  behavior  for 
command  and  control,  and 
non-govemmental  organizations. 

Principles  for  developing  physically 
and  cognitively  realistic  avatars; 

Principles  for  matching  the  level  of 
realism  in  human  behavior  and 
depiction  to  training  objectives; 

Accurate  and  facile  interchange  of  real 
and  virtual  participants;  Development 
of  sharable,  reusable  human  behavior 
objects;  Capabilities  that  allow  dynamic 
control  of  avatars  by  either  human 
instructors  or  automated  tutors. 

Instruction  Design 
and  Development: 

Representation  of 
knowledge  and  skill 

Automated  representation  of 
both  subject  matter  expertise 
and  misconceptions. 

Comprehensive  computer  representation 
of  human  knowledge  and  skill; 

Capabilities  for  automated  development, 
verification,  and  validation  of  these 
representations. 

Instruction  Design 
and  Development: 

Articulate  expertise 

Ability  to  inform  students  and 
aid  problem  solvers  in  terms  they 
are  prepared  to  understand. 

Ability  to  assemble  objects  and  prepare 
presentations  that  are  tailored  to  dynamic 
representations  of  the  student  or  problem 
solver. 

Instruction  Design 
and  Development: 

Natural  Language 
Understanding 

Ability  to  engage  in  natural 
language  dialogues  with 
students  and  problem  solvers. 

Speech  recognition;  Ability  to  parse 
natural  language;  Ability  to  parse 
dialogues;  Adequate  grammar  for  natural 
language  and  natural  language  dialogue. 
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Capability 

Description/Goal 

Requirements/Gaps 

Instruction  Design 
and  Development: 

Authoring  tools 

Fully  developed  capability  for 
automated  development  of 
instructional  and  performance 
aiding  materials  and  optimally 
efficient  tools  for  human  use 
when  automation  is  not 
practicable. 

Comprehensive  capability  for  generating 
instruction  and  performance  aiding 
materials  on-demand,  in  real-time, 
tailored  to  the  specific  needs  of 
individuals,  crews,  teams,  and  units; 

Ability  to  generate  instruction  and 
performance  aiding  automatically  from 
computer-aided  design  and  computer- 
aided  manufacturing  data  bases. 

Instruction  Delivery 
and  Management: 

Assessment  of 

leamer/user 

knowledge 

Ability  to  automatically  and 
unobtrusively  assess  student/ 
user  knowledge,  skills,  and 
needs  from  routine  interactions 
with  software. 

Computer  representation  of  user 
knowledge  skills,  and  needs; 

Generation  of  valid,  unobtrusive  near 
real  time  assessment  from  interactions 
of  individuals,  teams,  crews,  and  units 
with  the  learning  or  performance  aiding 
environment;  Automated  techniques  for 
creating  and  assessing  the  shared  mental 
models  of  crews,  teams,  and  units. 

Instruction  Delivery 
and  Management: 

Collaborative  team, 
crew,  and  unit  learning 

Ability  to  foster  optimal  leamer- 
to-leamer  collaboration  and 
leamer-to-instructor  interaction. 

Comprehensive  representation  of 
collaboration  and  interaction  in 
teams,  crews,  and  units;  Principles 
for  establishing  and  supporting 
collaboration  in  distributed  learning 
environments;  Capabilities  to  support 
in-route  mission  rehearsal  despite 
physical  separation  of  participants. 

Instruction  Delivery 
and  Management: 

Self-management  of 
instructional  progress 

Trainable  techniques  for 
developing  meta-cognitive 
capabilities;  Comprehensive 
tools  for  supporting  learner- 
directed  instruction  and  leaming- 
to-leam. 

Comprehensive  understanding  of 
meta-cognition  and  its  development; 
Comprehensive  understanding  of 
incentive  management  for  distributed 
learning;  Technology-based  tools 
allowing  distributed  learners  to  manage 
their  progress. 

Instruction  Delivery 
and  Management: 

Security  maintenance 

Multiple  levels  of  security  provided 
reliably  and  as  needed  across  a 
network  of  distributed  instruction 
and  performance  aiding  based  on  a 
pervasive  security  policy  and 
technical  strategy  that  addresses 
pan-network,  pan-organizational, 
authentication  and  data  access  and 
security. 

Reliable  learner/user  identification, 
authentication,  and  authorization; 

Standard  format  for  acquisition,  storage, 
maintenance,  retrieval,  and  application 
of  knowledge  bases;  Ability  to 
transparently  access  both  secure  and 
non-secure  sources  within  the  same 
network  for  both  training  and 
performance  aiding  purposes. 
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Capability 

Description/Goal 

Requirements/Gaps 

Instruction  Delivery 
and  Management: 

Integration  of 
approaches 

Integration  of  continuous, 
asynchronous  instruction  with 
current  classroom,  instructor-led 
approaches. 

Principles  for  combining  technology- 
based  with  human-based  instructional 
resources  and  approaches;  Development 
of  effective  organizational  structures. 

Instruction  Delivery 
and  Management: 

Instructional 

management 

Ability  to  optimize  student 
progress  toward  achieving 
instructional  or  performance 
goals. 

Ability  to  match  instructional  or  problem 
solving  goals  with  current  state  of  the 
student  or  user  and  generate  or  select 
optimal  tutorial  and/or  problem  solving 
strategies. 

Instruction  Evaluation: 

Cost  and  effectiveness 
analyses 

Fully  developed  system  for  cost- 
effectiveness  assessment  of 
education,  training,  and 
performance  aids  at  all  levels  to 
include  user  satisfaction, 
achievement  of  objectives, 
improvement  in  performance  at 
duty  stations,  and  enhancement 
of  readiness  and  effectiveness. 

Comprehensive  models  for  both  costs 
and  effectiveness  of  education,  training, 
and  performance  aiding;  Techniques  for 
linking  education  and  training  to  duty 
station  performance;  Verified  and  valid 
capabilities  to  assess  the  military  value 
of  education,  training,  and  performance 
aiding  in  enhancing  military  readiness 
and  effectiveness. 

How  Should  We  Get  From  Here  To  There?  A  Summary  in  a  NATO/PfP  Research  and 
Technology  Context 

Table  3  suggests  a  great  many  speeific  efforts  that  eould  and,  in  faet,  may  be  undertaken  eooperatively  by  the 
NATO/PfP  eommunity.  In  the  spaee  available  here,  we  ean  only  refer  the  reader  to  Table  3  as  a  list  of 
possibilities.  Nonetheless,  two  broad  areas  that  offer  speeifie  and  tangible  opportunities  deserve  mention. 
They  eoneem  the  sharing  of  information  through  a  eommonly  aecessible  data  base  and  the  development  of 
guidelines  and  standards  for  teehnology-based  edueation,  training,  and  performance  aiding. 

NATO/PfP  Database.  Some  of  the  efforts  suggested  by  Table  3  may  already  be  underway  and 
are  being  pursued  separately  and  independently  by  different  countries.  Even  though  these  efforts 
may  be  independent,  much  may  be  gained  by  sharing  experiences  and  findings  across  all 
countries  with  a  stake  in  their  outcomes.  Creation  of  a  NATO/PfP  directory  of  databases  could  be 
a  best  first  step.  This  database  would  point  to  data  elements  that  can  be  found  in  relevant 
databases  elsewhere  -  along  with  information  about  how  each  may  be  accessed.  Users  would 
engage  in  an  on-line  inquiry  to  focus  their  searches  on  (1)  the  specific  information  required  and 
(2)  the  identification  of  which,  if  any,  existing  databases  might  hold  the  information  they  are 
looking  for.  Retrieval  of  the  information  would  remain  in  users’  hands.  The  NATO  directory 
would  identify  what  data  elements  were  available  and  how  these  data  elements  might  be 
accessed,  but  the  next  steps  taken  would  be  up  to  the  users.  This  approach  would  describe  access 
to  all  data  elements  in  all  accessible  databases,  the  data  in  the  data  elements  would  be  kept  up  to 
date  by  those  who  ‘own’  the  database,  and  the  expenses  for  setup  and  maintenance  would  be 
minor.  This  approach  is  not  the  most  efficient  that  could  be  imagined,  but  it  represents  a 
beginning  and,  because  of  its  low  cost  and  high  feasibility,  might  increase  both  communication 
and  demand  for  more  efficiently  available  information. 
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NATO/PfP  Common  Practices.  The  second  opportunity  is  for  the  development  and 
establishment  of  common  practices  (if  not  standards)  for  the  development  of  sharable  objects  and 
guidelines  for  their  use  in  instructional  systems.  The  objects  would  be  intended  to  be  accessible 
on  the  global  information  grid  (today,  the  World  Wide  Web)  and  used  in  devising  military 
education,  training,  and  performance  aiding  presentations.  This  work  has  already  been  pursued 
by  organizations  in  the  United  States,  Europe,  and  the  Pacific  Rim.  Much  of  it  has  coalesced 
under  the  US  Department  of  Defense’s  Advanced  Distributed  Learning  Initiative,  which  has  been 
developing  the  Sharable  Courseware  Objects  Reference  Model  (SCORM)  (www.adlnet.org). 
Much,  however,  remains  to  be  done. 

Currently  these  efforts  have  focused  on  the  software  engineering  needed  to  create  objects  that  are 
accessible,  durable,  portable,  and  reusable.  Not  only  have  these  efforts  been  successful,  but  their 
prescriptions  are  likely  to  last.  They  have  been  developed  with  the  direct  support  of  those  in 
business  and  industry  who  are  most  likely  to  be  affected  by  them,  but  were  actively  seeking  a 
catalyst  (in  this  case  the  US  Department  of  Defense)  to  act  as  a  honest  broker  in  organizing, 
extracting,  and  documenting  a  set  of  compromises  resulting  in  the  standard  practices  they  all 
recognized  they  needed  if  the  market  for  their  software  tools  were  to  grow.  This  work  produced 
the  SCORM  model. 

But  how  are  these  objects  likely  to  be  used  in  real-world  applications  now  that  they  are  becoming 
available?  Guidelines  for  purchase  agents,  designers,  implementers,  evaluators,  and  even 
classroom  instructors  are  needed.  These  guidelines  must  be  produced  and  are  just  now  emerging. 
To  some  extent  they  will  respond  to  the  unique  needs  of  those  who  develop  them,  but  to  another, 
much  larger  extent,  standardization  in  these  guidelines  will  improve  efficiency,  reduce  costs, 
and  enhance  effectiveness  as  much  for  military  organizations  as  they  will  for  the  market  sought 
by  business  and  industry.  Such  standardization  in  practice  seems  a  natural  and  achievable  goal 
for  the  NATO/PfP  community. 


A  FINAL  WORD 

Human  competence  is  an  essential  part  of  every  military  system  and  every  military  operation.  Its  presence  will 
not  guarantee  the  full  performance  expected  from  every  military  system  nor  success  in  every  military 
operation,  but  its  absence  is  an  often  noted  cause  of  failure.  Few  activities  are  as  important  in  ensuring  the 
availability  of  human  competence  when  and  where  it  is  needed  as  are  education,  training,  and  performance 
aiding.  The  application  of  human-centered  technology  with  its  promise  for  reducing  costs  and  increasing 
operational  effectiveness  seems  both  an  economic  and  military  imperative.  The  NATO/PfP  research  and 
technology  community  has  the  opportunity  and  an  obligation  to  lead  its  members  into  the  future  promised  by 
these  technological  applications. 
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